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METHOD AND SYSTEM FOR THREE-DIMENSIONAL IMAGING USING 
LIGHT PATTERN HAVING MULTIPLE SUB-PATTERNS 

REFERENCE TO RELATED APPLICATIONS 
[0001] This application claims the benefit of U.S. Provisional Appln. No. 60/178,695, 
filed January 28, 2000, the disclosure of which is incorporated herein by reference in its 
entirety. 

BACKGROUND OF THE INVENTION 

Field of the Invention 

[0002] The present invention is directed to three-dimensional surface profile imaging, 
and more particularly to a method and apparatus for three-dimensional imaging that uses 
color ranging to conduct surface profile measurement. 

Description of the Related Art 

[0003] A three dimensional surface profile imaging method and apparatus described in 
U.S. Patent No. 5,675,407 ("the '407 patent"), the disclosure of which is incorporated 
herein by reference in its entirety, conducts imaging by projecting light through an optical 
filter, such as linear variable wavelength filter (LVWF), thereby projecting light having a 
known, spatially distributed wavelength spectrum on the objects being imaged. The 
LVWF is a rectangular optical glass plate coated with a color-filtering film that gradually 
varies in color, (i.e., wavelength). If the color spectrum of a LVWF is within the visible 
light region, one edge of the filter rectangle may correspond to the shortest visible 
wavelength (i.e. blue or violet) while the opposite edge may correspond to the longest 
visible wavelength, (i.e. red). The wavelength of light passing through the coated color- 
filtering layer is linearly proportional to the distance between the position on the filter 
glass where the light passes and the blue or red edge. Consequently, the color of the light 
is directly related to the angle G, shown in Figure 1, at which the light leaves the rainbow 
projector and LVWF. 
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[0004] Referring to Figure 1 in more detail, the imaging method and apparatus is based 
on the triangulation principle and the relationship between a light projector 100 having 
the LVWF, a camera 102, and the object being imaged 104. As shown in Figure 1, a 
triangle is uniquely defined by the angles theta (0) and alpha (a), and the length of the 
baseline (B). With known values for 9, a, and B, the distance (i.e., the range R) between 
the camera 102 and a point Q on the object's surface can be easily calculated. Because 
the baseline B is predetermined by the relative positions of the light projector 100 and the 
camera 102, and the value of a can be calculated from the camera's geometry, the key to 
the triangulation method is to determine the projection angle, 9, from an image captured 
by the camera 102 and more particularly to determine all 0 angles corresponding to all 
the visible points on an object's surface in order to obtain a full-frame 3D image in one 
snapshot. 

[0005] Figure 2 is a more detailed version of Figure 1 and illustrates the manner in which 
all visible points on the object's surface 104 is obtained via the triangulation method. As 
can be seen in the Figure, the light projector 100 generates a fan beam of light 200. The 
fan beam 200 is broad spectrum light (i.e., white light) which passes through the LVWF 
to illuminate one or more three-dimensional objects 104 in the scene with a pattern of 
light rays possessing a rainbow-like spectrum distribution. The fan beam of light 200 is 
composed of multiple vertical planes of light, or "light sheets", each plane having a given 
projection angle and wavelength. Because of the fixed geometric relationship among the 
light source 102, the lens of the camera 102, and the LVWF, there exists a one-to-one 
correspondence between the projection angle (0) of the vertical plane of light and the 
wavelength (k) of the light ray. Note that although the wavelength variations are shown 
in Figure 2 to occur from side to side across the object 104 being imaged, it will be 
understood by those skilled in the art that the variations in wavelength could also be 
made from top to bottom across the object 104 being imaged. 

[0006] The light reflected from the surface of the object 104 is then detected by the 
camera 102. If a visible spectrum range LVWF (400 - 700 nm) is used, the color 
detected by the camera pixels is determined by the proportion of its primary color Red, 
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Green, and Blue components (RGB). The color spectrum of each pixel has a one-to-one 
correspondence with the projection angle (G) of the plane of light due to the fixed 
geometry of the camera 102 lens and the LVWF characteristics. Therefore, the color of 
light received by the camera 102 can be used to determine the angle 9 at which that light 
left the rainbow light projector. Other spectrum ranges can also be used in similar 
fashion. The implementation is straightforward to those skilled in the art. 

[0007] As described above, the angle a is determined by the physical relationship 
between the camera 102 and the coordinates of each pixel on the camera's imaging plane. 
The baseline B between the camera's 102 focal point and the center of the cylindrical lens 
of the light projector 100 is fixed and known. Given the value for angles a and 9, 
together with the known baseline length B, all necessary information is provided to easily 
determine the full frame of three-dimensional range values (x,y,z) for any and every 
visible spot on the surface of the objects seen by the camera 102. 

[0008] As shown in Figure 3, given the projection angle 9, the three-dimensional 
algorithm for determining the (x,y,z) coordinates of any surface spot Q(x,y,z) on a three- 
dimensional object is given below based on the following triangulation principle: 

B B B 

x = * w, y = * v, z = * / (1) 

f*ct g e-u J f*ct g e-u f*ct g e-u 

[0009] As a result, the three-dimensional imaging system described above can capture 
full-frame, high spatial resolution three-dimensional images using a standard camera, 
such as a charge coupled device camera, in real time without relying on any moving 
parts. Further, because the imaging system does not rely on a laser, it does not pose any 
hazard to the eyes when used in clinical applications. Also, because the wavelength of 
the light projected onto the object surface continuously varies, there is no theoretical 
limitation on the measurement accuracy that can be achieved by the system. The actual 
accuracy of a specific system will depend on system implementation and will be affected 
primarily by limiting factors such as the optical system design, the quality and resolution 
of the camera, light spectral emission of the light source projector; noise level and 
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resolution of the frame grabber, calibration algorithms, and the three-dimensional 
imaging processing algorithms. 

[0010] To avoid allowing the ambient light on the object being imaged from affecting the 
imaging results, the system may obtain an image of the object under normal light 
conditions before projecting the filtered light onto the object. The image obtained under 
normal light conditions is then subtracted from the image obtained under LVWF light 
conditions to eliminate the effects of the ambient light on the image. 
[0011] Referring back to Figure 1, the triangulation algorithm used in the imaging system 
is based on the following formula, with reference to Figure 3: 

R = ^B, (2) 
sin a 

where (x p , y p ) is the location of the rainbow light projector, (x c , y c ) is the location of 
imaging sensor, B is the baseline between the rainbow projector and the imaging sensor 
(CCD), a = 7t - 9 -p\ Oisa surface point on the object in the scene, and R is the three- 
dimensional range, that is, the distance between (x c , y c ) and O. 

[0012] Note that all of the variables, 0, a, and p\ in the equation 10 may introduce error 
in the three-dimensional range calculation. In the following error sensitivity analysis, 
considered with reference to Figure 4, it is assumed that the coordinate of the camera's 
focal point in the world coordinate system can be obtained precisely through camera 
calibration. The full derivative of the R is given by: 

r sin<9cos# ... , r cos6', 1 ,_ sin<9 ^ 

dR = [ = b]da + [ b]dO + [ ]db (3) 

-sin a sina sma 



The da, d0, and db are all functions of the position error of the light projector (dx p , dz p ) 
and the color mismatch dx. Notice that: 



b = ^{x c -x s f +{z c -z s f , 




1 z r —Z„ , Z„ ,,s 

0 = tan- x — ^ + tan _1 p — (4) 

x c -x p x-x p 
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(x -x c )dx + (z - z c )dz p 
db = 



dO-- 



d p = ( x cf- x p) dz P + ( z p- z c') dx P (5) 
b 2 

(x p -x c )dz p + (z c -z p )dx p z p 



b 2 (x-x p ) 2 +z p 



[0013] To simplify the analysis, the effect of a mismatch error in the world coordinate, 
dx, caused by imaging sensor's spectral noise can be indicated as follows: 

dR = ] ^l b] (6) 

siner (x-x p ) +z p 

[0014] The linear relationship between the mismatch error dx in the world coordinate and 
the mismatch error in CCD camera's pixel coordinate, di, allows approximation of dx and 
di as if they are the same order of magnitude, resulting in the expression: 

dx = k * di ( 7 ) 
where k is a constant. From this expression, the range error in terms of the color 
mismatch error in CCD pixel coordinate can be expressed as follows: 

dR = [^b] \-^*k*di (8) 

sina {x-Xp) +z p 

[0015] In the imaging system shown in Figures 1 and 2, the continuously varying 
wavelength pattern obtained via the light projector 100 and the LVWF 101 is projected 
spatially onto surfaces of a three-dimensional object or scene 104. There is a one-to-one 
corresponding relationship between the wavelength of the projected light sheet, co, and its 
projection angle, 9. Based on this one-to-one relationship, a (©,9) lookup table can be 
established for the color matching operation. An image sensor device (such as the CCD 
camera 102) that collects the reflection from object's surfaces also provides wavelength 
information for each pixel in an image, enabling the determination of the projection angle 
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0 corresponding to each image pixel through the color matching operation based on the 
(co,9) lookup table. 

[0016] The accuracy of the color match operation, of course, affects the accuracy of the 
three-dimensional surface measurement. Referring to Figure 5, the accuracy of the color 
match is significantly determined by the characteristics of the projected wavelength- 
varying illumination pattern. More particularly, the system has a signal to noise ratio 
(S/N = c), where the noise is a catch-all term that characterizes all undesirable effect in 
the system to reduce the fidelity of the spectral signal (i.e., wavelength). Due to the 
existence of the noise, mismatch error (denoted as dw) occurs. 

As shown in Figure 6, for the same magnitude level of dw, if we use a spatially varying 
projection pattern that has a greater derivative, a smaller mismatch error in the pixel 
coordinate, denoted as di, occurs due to the increased slope caused by the larger 
derivative: 

di 2 <dii. (9) 
Using Equ. (8) in Equ. (9), the expression becomes: 

dR 2 <dRi. (10) 



The above expressions suggest that mismatch errors can be reduced by using a spatially 
varying projection pattern that has a large variation rate and therefore a large derivative. 

[0017] As noted above, the imaging system projects a light pattern that could be 
described as a "single rainbow projection" pattern because the pattern ranges over the 
visible light spectrum a single time. This type of pattern, however, has a variation rate 
that depends on numerous factors, including the field of view of the camera/image 
sensor, the distance between the light projector and the object being imaged, as well as 
other factors. As a result, it is difficult to adjust the spectral variation rate of the single 
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rainbow projection pattern to fit the accuracy requirements of many three-dimensional 
imaging tasks. 

[0018] There is a need for a method and system that can optimize the light pattern to 
provide higher measurement accuracy. 

SUMMARY OF THE INVENTION 

[0019] Accordingly, the present invention is directed to a system and method that 
generates a light pattern that contains multiple cycles of light spectra or sub-patterns. In 
one embodiment, the light pattern includes two or more cycles of spectra of visible or 
infrared light, arranged contiguously with each other. The light pattern may also be 
generated by sequentially emitting two or more sub-patterns. 

[0020] In another embodiment of the invention, the multiple spectra light pattern is 
generated by projecting the light from the light source through a planar member, such as 
a slotted panel, that is movable to generate small phase shifts in the light pattern. The 
multiple patterns and/or sub-patterns used to illuminate the object being imaged improves 
resolution and measurement accuracy by reducing color mismatch error. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0021] Figure 1 is a simplified diagram illustrating a triangulation principle used in the 
present invention; 

[0022] Figure 2 is a representative diagram of the components used by the inventive 
system; and 

[0023] Figure 3 is a plan view of the system shown in Figure 2; 

[0024] Figure 4 is a representative diagram illustrating the effect of errors in range 

calculation; 
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[0025] Figures 5 a and 5b are graphs comparing the slopes of the illumination pattern on 
measurement accuracy; 

[0026] Figures 6a and 6b are light pattern spatial displacement graphs corresponding to 
one embodiment of the present invention; 

[0027] Figures 7a, 7b and 7c illustrate a multiple spectra light projection pattern 
according to the present invention; 

[0028] Figure 8 is a representative diagram of the manner in which a point search is 
conducted for the multiple spectra light projection pattern; 

[0029] Figure 9 is a flowchart of an adaptive initial point calculation method used in the 
present invention; 

[0030] Figures 10a and 10b are diagrams illustrating the effect of sensor characteristics 
on light pattern distortion; 

[0031] Figure 11 illustrates another embodiment of the inventive system; 

[0032] Figures 12a and 12b illustrate a light pattern and method that is generated from a 

composite image of multiple sub-patterns according to another embodiment of the 

invention; 

[0033] Figure 13 is a representative diagram of another embodiment of the invention 
using a slotted plate; 

[0034] Figure 14 illustrates examples of cross-sections of the slotted plate used in Figure 
13; 

[0035] Figure 15 is an illustrative diagram showing the manner in which the slot plane 
generates a spatially varying light intensity pattern; and 

[0036] Figures 16a and 16b illustrate alternative systems incorporating the slotted plate. 
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15. The device of claim 1, wherein the varying characteristic is a spatially 
varying intensity and the device further comprises a planar member having a plurality of 
openings disposed between said light source and said object or scene for generating the 
light pattern. 

16. The device of claim 15, wherein said plurality of openings are slots. 

17. The device of claim 15, further comprising a microshifting mechanism 
coupled to the planar member to generate shifts of the light pattern by moving the planar 
member, wherein the processor calculates a distance to the point on the object or scene 
using triangulation based on the shifts of said light pattern. 

18. The device of claim 17, wherein the microshifting mechanism includes a 
support structure coupled to the planar member, wherein the support structure is formed 
of a bimorpher material that is movable in response to an applied voltage to thereby move 
the planar member. 

19. The device of claim 17, wherein the microshifting mechanism includes a 
support structure coupled to the planar member and an electromagnetic actuator to move 
the support structure and thereby move the planar member. 

20. A device for creating a three-dimensional profile of an object or scene 
being imaged, the device comprising: 

a light source for illuminating said object or scene with a light pattern, wherein 
said light pattern has a spatially varying characteristic that varies across said object or 
scene and includes two or more visible light spectra arranged contiguously with each 
other such that light having a given characteristic is emitted at two or more projection 
angles; 

an image sensor for imaging said object or scene as illuminated with said light 
pattern; and 
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a processor for calculating a distance to a point on said object or in said scene 
based on a baseline distance between said light source and said camera, an angle between 
said camera and said baseline, and an angle at which light striking the point is emitted by 
said light source as determined from the given characteristic of the light striking the 
point. 

21 . The device of claim 20, wherein the spatially varying characteristic is 
wavelength, and wherein the processor selects from said two or more projection angles 
by correlating the point position with one of said two or more visible light spectra. 

22. The device of claim 21, wherein the processor determines the projection 
angle for the given wavelength via adaptive initial point calculation, wherein the 
processor restricts a search space for color matching in one of said visible light spectra. 

23. The device of claim 20, wherein the image sensor is pre-calibrated to 
generate a pre-distorted projection pattern. 

24. The device of claim 20, further comprising a second image sensor, 
wherein said image sensor and said second image sensor together form a stereo pair. 

25. The device of claim 20, wherein the spatially varying characteristic is 
wavelength, and wherein the device further comprises a planar member having a plurality 
of openings disposed between said light source and said object or scene for generating the 
light pattern. 

26. The device of claim 25, further comprising a microshifting mechanism 
coupled to the planar member to generate shifts of the light pattern by moving the planar 
member, wherein the processor calculates a distance to the point on the object or scene 
using triangulation based on the shifts of said light pattern. 
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27. The device of claim 26, wherein the micro shifting mechanism includes a 
support structure coupled to the planar member, wherein the support structure is formed 
of a bimorpher material that is movable in response to an applied voltage to thereby move 
the planar member. 

28. The device of claim 26, wherein the microshifting mechanism includes a 
support structure coupled to the planar member and an electromagnetic actuator to move 
the support structure and thereby move the planar member. 

29. A method for creating a three-dimensional profile of an object or scene 
being imaged, the method comprising the steps of: 

illuminating the object or scene with a light pattern, wherein said light pattern has 
a spatially varying characteristic that varies across said object or scene and includes two 
or more sub-patterns; 

imaging said object or scene as illuminated with said light pattern; and 
calculating a distance to a point on said object or in said scene based on a baseline 
distance between said light source and said camera, an angle between said camera and 
said baseline, and an angle at which light striking the point is emitted by said light source 
as determined from the spatially varying characteristic of the light striking the point. 

30. The method of claim 29, wherein the spatially varying characteristic is 
wavelength, and wherein the two or more sub-patterns are two or more spectra of light in 
the visible light spectrum such that a given wavelength corresponds to more than one 
projection angle. 

31. The method of claim 30, wherein the calculating step includes the step of 
determining the projection angle for the given wavelength via adaptive initial point 
calculation by restricting a search space for coloring matching in one of said sub-patterns. 
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32. The method of claim 29, wherein the spatially varying characteristic is 
intensity, and wherein the two or more sub-patterns from the illuminating step are three 
light sub-patterns corresponding to red, green and blue components of the light pattern, 
and wherein the three light sub-patterns have varying intensities, each sub-pattern having 
a high point and a low point such that the high points and low points for the three sub- 
patterns are distributed over a spatial period of the light pattern. 

33. The method of claim 29, further comprising the step of pre-distorting the 
light pattern based on a pre-calibration characteristic of the image sensor. 

34. The method of claim 29, wherein the imaging step comprises the step of 
imaging the illuminated object or scene using at least two image sensors. 

3 5 . The method of claim 29, wherein the spatially varying characteristic is 
intensity, and wherein the illuminating step comprises the step of generating the light 
pattern by sequentially emitting a plurality of light sub-patterns corresponding to red, 
green and blue components of the light pattern. 

36. The method of claim 29, further comprising the steps of shifting the light 
and calculating a distance to the point on the object or scene using triangulation based on 
the shifts of said light pattern. 
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where s is the location of the point light sources element, / is the location on the surface 
of object, I(s, I) is the illumination intensity on the surface of object at the / location 
produced by point light source of s. Accumulation of the light illumination from all of 
the point light sources forms the desired spatially varying illumination intensity on the 
object's surface at locations that are away from the focal plane of the light projection 
system, as illustrated in Figure 15. Shifting the position of the slotted plate 1002 relative 
to the object surface creates small phase shifts of the illumination intensity pattern to 
allow triangulation of each point on the object being imaged. 

[0055] Figures 16a and 16b illustrate two possible structures for shifting the slot plate 
position to provide phase shifts in the illumination intensity pattern. Figure 1 6a shows 
one possible microshifting mechanism using a piece of bimorpher material 1004 to 
support the slotted plate 1002. Bimorpher materials have the unique property of 
electrical field induced motion. By fixing one end of the bimorpher support 1004, the 
other end of the bimorpher support 1004 can be controlled to move transversally by 
applying DC voltage on the two sides of the bimorpher support 1004. The bimorpher 
support 1004 can be fixed, for example, on the light source and the slot plate, supported 
by the bimorpher support 1004, can be controlled to move the slot plate 1002 relative to 
the position of the light source 100 by applying different electrical fields to the bimorpher 
support member 1004. Consequently, the zigzag illumination pattern can be shifted. In 
many application of three-dimensional imaging, the micro-shifting of the slot plate 1002 
position can generate sufficient shift of the zigzag waveform to allow for triangulation of 
each point on the object being imaged 104. 

[0056] An alternative embodiment, as shown in Figure 16b, of the micro-shift 
mechanism uses a pair of miniature electromagnetic actuators 1004 and a flexible bar 
1005 supporting the slotted plate 1004. More particularly, activation of the actuators 
1004 can move the flexible bar 1005, and thereby the slot plate 1002, in transverse 
directions. 

[0057] The slotted plate 1002 can be used with, for example, a camera and a single-color 
light source. In such a case, the monochromatic camera preferably illuminates the object 
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three times with the slotted plate 1002 in a different position each time. These three 
images are then processed by the system's computer as if they represented the red, green 
and blue components of a color image. The resulting composite image, while not 
actually in color, contains variation that can be arbitrarily associated with color. This 
pseudo-color image is then used by the system to determine the angle at which light left 
the light source to illuminate each point or pixel on the object being imaged. This 
method is proceeds in precisely the manner outlined above when, for example, an LVWF 
is used to provide a rainbow or wavelength varying light pattern on the object being 
imaged. In other words, after the three images are taken between shifts of the slotted 
plate, these images being used as red, green and blue components of a composite color 
image, and a pseudo-color composite image is constructed, three-dimensional ranging 
can be performed according to any ranging technique such as, for example, the technique 
described in U.S. Patent No. 5,675,407, the disclosure of which is incorporated herein by 
reference. 

[0058] While the invention has been specifically described in connection with certain 
specific embodiments thereof, it is to be understood that this is by way of illustration and 
not of limitation, and the scope of the appended claims should be construed as broadly as 
the prior art will permit. 
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